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IDENTIFIERS 
ABSTRACT 

One hundred eighty limited-English-speaking college 
students from the Philippines, Bong Kong, India, Japan, Thailand, and 
Taiwan were administered the Michigan Test of Aural Comprehension 
(HTAC). An analysis of the MTAC indicated that the three unrestricted 
forms of the test shcj satisfactory internal consistency reliability 
comparable to that found for other listening comprehension tests. 
However, the results strongly suggest that counselors should not 
treat the three forms as being statistically eguivalent. Furthermore, 
there is strong evidence that speaker variation, contrary to the test 
manuals* assertions, can effect test results. <Author/£W) 
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ABSTRACT 



An examination of test results for ESL and EFL college students 
from six countries for Forms A, B, and C of the Michigan Test of Aural 
Comprehension provided test reliability estimates previously unavailable. 
Further analysis indicated a lack of equivalence between forms and the 
need for caution in selecting test administrators wi th di fferent English 
language accents. 
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THE RELIABILITY AND INTERRELATIONSHIPS OF 
THREE FORMS OF THE MICHIGAN TEST OF AURAL COMPREHENSION 



Oral comprehension is an essential element in the repertoi ra of 
skills necessary for the non-native-English-speaking applicant who 
wants to study at an institution of higher education in the United 
States (Smith & Bisazza, submitted). 

A glance at Buros (1978) indicates that there are few non-secure 
tests the counselor can use to evaluate initial or progressive develop- 
ment of aural comprehension skills of such students. One test which is 
available Is the Michigan Test of Aural Comprehension (MTAC) (Lado, 
1957). 

The MTAC is a ^0-minute test designed to measure 'understanding of 
spoken English by persons whose native language is not English 1 (Lado, 
1957, p.1). The test seems suitable for evaluation studies because 
three parallel non-secure forms are available. However, there is little 
empirical evidence in the test manual or in the literature about the 
reliability of the MTAC, or the interrelationship of the three forms of 
the test. John B. Carroll in a review of the tests noted that the manual 
'claims that the test scores from the three forms are equivalent, but 
gives no information concerning equivalent test reliabilities or the 
like 1 (Buros, 1978, p.188). This study was undertaken to collect such 
data using students from six countries, three where English is a second 
(ESL) and three where English is a foreign (EFL) language. 
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Methods 

Subjects 

The subjects, 30 each from the Philippines, Hong Kong, India, 
Japan, Thailand, and Taiwan, were enrolled in beginning English 
language university programs. The six groups were roughly balanced 
for sex and age and all had had at least five years of h ior English 
language study. In the first three countries mentioned, English is 
spoken as a second language while in the latter three, English is a 
foreign language. 
Instruments 

The MTAC is a two nart AO-minute examination with three concep- 
tually equivalent forms, each of which consists of 60 sentences and 
short paragraphs to be read aloud by an examiner. Students respond 
using a booklet containing three pictures or phrases, each of which is 
the correct answer to a question for one of the three forms. The 
study used the three earliest forms of the MTAC since they are not 
currently restricted and are being used in Asia and the Pacific. 

The test manual claims thot forms A, B, and C are of approximately 
equal difficulty. An alternate-forms-reliability between forms A and 
C of .87 and a split-half reliability of .88 for form A are given. No 
norms are available, but proficiency and progress standards based on 
percent correct scores are provided. 
Procedures 

The design of the study uses a complex latin squares design pro- 
cedure (Bruning and Kintz, 1977) • The three forms of the test were 
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given on separate days within a ten-day period to studercs in each of 
the six countries. Three variables, order of presentation, test form, 
and speaker were balanced within each of the three countries in the 
ESL and EFL groups. Taped presentations of the three proficient 
speakers of English with different accents (native, Indian ESL, and 
Japanese EFL) were used to ensure a standardized administration of the 
test materials. 

' Results 

Table 1 provides Kuder-Ri chardson (KR) formula 20 estimates of 
internal consistency and their corresponding standard errors of esti- 
mate by English language status and country for percentage correct 
scores from forms A, B, and C of the MTAC. Alternate forms reli- 
ability estimates based on Pearson r^'s were calculated between three 
test forms. 



INSERT TABLE 1 HERE 



The data were first analyzed to see if there was an order of 
presentation effect. A significant percentage score increment of 2.5 
points was found between both pai r s of test administration neans. To 
control for this effect, T-scores were generated for each order of 
presentation. The resulting data, with order of presentation now 
controlled, was then analyzed using a complex latin squares design 
which balanced test form (A, B, C) and type of speaker (native, ESL, 
EFL), within each language status (ESL, EFL) group. These results are 
presented in Table 2. 
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INSERT TABLE 2 HERE 



Discussion 

As Table 1 Indicates, the internal consistency estimates found in 
this study averaged in the mid to upper .70's with a standard error of 
about 4.50. While these values are lower than the split-half reli- 
ability of .88 with a standard error of .02 (surely this latter figure 
is incorrect) reported in the manual, they are consistent w"th the 
somewhat lower reliabilities generally associated with listening 
comprehension tests. The standard error of estimate of about 4.50 for 
the percentage scores should provide test users with a more accurate 
estimate for evaluating student score differences. As the following 
analysis indicates, the lower median test-retest reliabilities of .54, 
.57, and .64 may be due in part to the confounding effects created by 
the design of the study when significant variables effects are found. 

The 1 .< 1 1 n squares analysis (Table 2) indicates that the three 
forms of the test are not statistically equivalent (F 2 ,348 = 47.89, 
p<.00l). Scores on form A averaged one quarter of a standard 
deviation beiv>/ those for form C, while scores on form B were nearly 
half a standard deviation below the scores on form C. Furthermore, 
the significant form by type interaction (F 2 , 3^8 ■ 35.18, p<.00l) 
indicates that these differences were not consistant for ESL f;»d EFL 
students* On form A ESL student scores averaged 1.3 standard 
deviations abo/e EFL students. Student scores for form B were only 
1.0 standard deviation apart while scores on form C were 0.55 standard 
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deviations apart. These results strongly suggest that alternate forms 
pre-post test designs should be avoided, especially with heterogeneous 
groups . 

The manual states that 'Any teacher with a reasonably clear voice 
who speaks a variety of Standard English can give the test. 1 (Lado, 
1957, p. 2). However, the speaker effect found in this study (F 2 , 3^8 = 
102.16, p<.00l) strongly suggests this is not the case. Students 
averaged 0.21 standard deviatic-s better on the test when it was given 
by a native speaker than when it was given by a fluent EFL speaker 
from Japan. When the tests were given by a fluent ESL speaker from 
India, they averaged 0.6** of a standard deviation below those results 
obtained by the native speaker. Speaker variation clearly effects 
test results, at least when the presentation is made using tape 
recorded presentations. Test designs should employ the same or a 
comparable speaker to avoid biasing test results. 

Conclusions 

An analysis of the MTAC indicated that the three unrestricted 
forms of the test show satisfactory internal consistency reliability 
comparable to that found for other listening comprehension tests. 
However, the results strongly suggest that counselors should not treat 
the three forms as being statistically equivalent. Furthermore, there 
is strong evidence that speaker variation, contrary to the test 
manuals' assertions, can effect test results. 
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TABLE 1 



Internal Consistency, Standard Errors, and AUernate 
Forms Reliability Estimates for the Michigan Test 
of Aural Comprehension by Language Status and Country 



. . fc KR Internal Consistency Alternate Forms 

Language Status 21 



Country 


N 








SE M 


Ice 


SE M 








:SL Students 






















Hong Kong 


30 


.78 


3.80 


.81 


3.42 


.74 


4.32 


.70 


.48 


.33 


India 


30 


.92 


4.32 


.91 


4.55 


.92 


4.11 


.77 


.85 


.74 


Phi lippines 


30 


.78 


2.68 


.61 


4. 11 


.46 


2.83 


.34 


.17 


.18 


■FL Students 






















Japan 


30 


.78 


4.69 


.62 


4.84 


.64 


4.16 


.54 


.49 


.18 


Taiwan 


30 


.59 


4.54 


.76 


4.35 


.78 


4.56 


.57 


.65 


.75 


Thai land 


30 


.68 


4.93 


.75 


4.90 


.76 


4.48 


.73 


.74 


.71 



Medi an 

Values .78 4.1»3 .76 4.45 .75 4.24 .64 .57 .54 
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TABLE 2 

Complex Latin Squares ANOVA for Test Form by 
Speaker by Student Type 



Source 


SS 


df 


MS 






F 


Between Subjects 


I, A A 1. A A A 

HD t 3H\) .00 


179 










Student Type 


10 O O 1 OA 
12,231 .30 


1 


12,231. 


30 


106 


1 


Speaker X Forn^ 


"7 ACQ C fi 


*> 

2 


3,520. 


77 


30 


£ 1. JLJL A 


1 AnL 1A 

1 | DUM • IO 


9 

L 


CUZ • 


uo 


L 
D 




Error. 

D 


20,0l»5.00 


m 


115. 


20 






Within Subjects 


13,52i».98 


360 










Speaker 


3.827.M 


2 


1,913. 


70 


102 


.16*** 


Form 


1,79*. 1* 


2 


897. 


07 


*7 


.89**" 


Speaker X Form^ 


0.2h 


2 


0. 


12 


0 


.01 


Speaker X Type 


61.23 


2 


30. 


61 


1 


.63 


Form X Type 


1,318.27 


2 


659. 


14 


35 


. 18*** 


Spk X Form^ X Type^ 


i».38 


2 


2. 


69 


0 


.14 


Error 

w 


6,519.31 


3^8 


18. 


73 







Total 54^64.98 539 



Those sources with mixed between and within components have been 
tested at the alpha/2 percent level. 

** p<01 
*** p<.001 
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